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ABSTRACT 

Objective Inadequate participant recruitment is a major 
problem facing clinical research. Recent studies have 
demonstrated that electronic health record (EHR)-based, 
point-of-care, clinical trial alerts (CTA) can improve 
participant recruitment to certain clinical research studies. 
Despite their promise, much remains to be learned about 
the use of CTAs. Our objective was to study whether 
repeated exposure to such alerts leads to declining user 
responsiveness and to characterize its extent if present to 
better inform future CTA deployments. 
Methods During a 36-week study period, we 
systematically documented the response patterns of 178 
physician users randomized to receive CTAs for an 
ongoing clinical trial. Data were collected on: (1) response 
rates to the CTA; and (2) referral rates per physician, per 
time unit. Variables of interest were offset by the log of 
the total number of alerts received by that physician 
during that time period, in a Poisson regression. 
Results Response rates demonstrated a significant 
downward trend across time, with response rates 
decreasing by 2.7% for each advancing time period, 
significantly different from zero (flat) (p<0.0001). 
Even after 36 weeks, response rates remained in the 
30%— 40% range. Subgroup analyses revealed 
differences between community-based versus 
university-based physicians (p=0.0489). 
Discussion CTA responsiveness declined gradually over 
prolonged exposure, although it remained reasonably 
high even after 36 weeks of exposure. There were also 
notable differences between community-based versus 
university-based users. 

Conclusions These findings add to the limited literature 
on this form of EHR-based alert fatigue and should help 
inform future tailoring, deployment, and further study of 
CTAs. 
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BACKGROUND AND SIGNIFICANCE 

Clinical trials are essential to the advancement of 
medicine, and research participant recruitment is 
critical to successful trial conduct. Unfortunately, 
difficulties achieving recruitment goals are 
common, and failure to meet such goals can impede 
the development and evaluation of new medical 
therapies.^ ^ It is well recognized that physicians 
often play a vital role in the recruitment of 
participants for certain trials. However, barriers 
including time constraints, unfamiliarity with 
available trials, and difficulty referring patients to 
trials, often make it challenging to recruit during 
routine practice. Consequently most clinicians 



do not engage in traditional recruitment activities 
and recruitment rates suffer^ * 

The increasing availability of electronic health 
records (EHRs) presents an opportunity to address 
the issue of inadequate recruitment for clinical trials 
by leveraging the information and decision support 
resources often built into such systems. Indeed, 
recent studies of EHR-based, point-of-care, clinical 
trial alerts (CTA) have demonstrated they have the 
potential to improve recruitment rates when 
applied to clinical trials.^^' Despite their promise 
and the fact that they have been well tolerated in 
recent studies, CTAs like any point-of-care alert do 
have the potential for misuse, and further study is 
needed to better understand and inform their 
appropriate, widespread use.^" One important but 
poorly understood aspect relates to the performance 
characteristics of such alerts, particularly the issue 
of clinician responsiveness to alerts over time, and 
the implications of such phenomena on alert design 
and deployment decisions. 

It is well recognized that when clinicians are 
exposed to too many clinical decision support (CDS) 
alerts they may eventually stop responding to them. 
This phenomenon is often called alert fatigue.^' 
While definitions vary and empirical evidence as to 
its cause is limited, alert fatigue is generally thought 
to result from one or more distinct but closely 
related factors. One such factor is declining clinician 
responsiveness to alerts as the number of simulta- 
neous alerts increases. This is thought to be related 
to issues such as alert irrelevance and cognitive 
overload, and has also been referred to as 'alert 
overload.''^ A second factor relates to declining 
clinician responsiveness to a particular type of alert 
as the clinician is repeatedly exposed to that alert 
over a period of time, gradually becoming 'fatigued' 
or desensitized to it.^^ Few studies have explored 
this latter issue of 'fatigue' due to repeated exposure 
to alerts over time,''* and it is this latter aspect of 
alert fatigue that motivated and was the focus of 
the current study. 

Although the purpose of CTAs is to provide 
decision support for trial recruitment rather than for 
clinical care, we hypothesized that this phenomenon 
of alert fatigue over time would be a factor in the 
usage patterns of CTAs, as also noted for CDS alerts, 
and would present itself as a gradual reduction in 
response rates over time. As the nature and extent 
of this issue and the overall performance character- 
istics of CTAs are not fully understood, the current 
study was performed among physician subjects as a 
planned part of a recently conducted randomized 
controlled intervention study of a CTA.'^ 
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METHODS 

We performed a cluster-randomized controlled study of a CTA 
intervention across three health system environments that share 
a common, commercial ambulatory EHR (GE Centricity EMR). 
The study of the CTA and the clinical trial to which it was 
applied were approved by our institutional review board. The 
associated clinical trial involving patients with insulin resistance 
after a recent stroke event was registered in Clinical Trials.gov 
(NCT00091949). 

Subjects involved in this 36-week study of the CTA inter- 
vention in 2009 (the first, pre-cross-over phase of a larger study 
on the impact of this CTA intervention) included 178 physicians 
who were randomly divided into equal groups within their 
specialties (ie, neurologists, n=26; family medicine physicians, 
n=35; general internists, n=46; internal medicine-pediatrics 
specialists, n=8; and internal medicine house staff, n=63). All 
neurologists were university-based practitioners, while the other 
generalist physicians practiced either in university-based or 
community-based settings. Prior to CTA activation, all physi- 
cians were encouraged via traditional means (eg, discussion at 
staff meetings, email blasts, flyers) to recruit patients to the 
trial. 

Upon CTA activation, intervention physicians seeing eligible 
patients were presented with on-screen CTAs that suggested 
they consider and discuss trial recruitment with the patient, and 
click an on-screen button to send a secure referral message to the 
trial coordinator if appropriate (figure 1). The physicians 
involved in this study did not receive any incentives for their 
participation in these recruitment efforts. It is worth noting that 
the results of the phase of the underlying CTA intervention 
study during which this analysis of responses took place revealed 
a significant 20-fold increase in referrals (p<0.0002) and a nine- 
fold increase in enrollments (p<0.006).^^ The design of the CTA 
intervention itself has also previously been reported and 
involved a minimal amount of novel programming as the built- 
in tools and resources of the particular EHR system were used.*^ 
This was a similar approach to that employed with another 
popular EHR system upon which we have previously reported.*' 

Additional data were also collected during the same period via 
direct query of the EHR-fed enterprise data warehouse for the 



two types of events in which we were interested: (1) responses 
that indicated interaction with the CTA (ie, taking action by 
responding to at least one of the questions posed in the alert); and 
(2) flags (referrals) sent to the study coordinator via positive 
responses to both questions and then processing of the CTA. In 
addition, data on the total number of alerts triggered were 
extracted as the denominator value for determination of a ratio of 
(1) response rate and (2) referral rate per physician, per time unit. 

In our analyses, the dependent variables 'responses' and 
'referrals' were offset by the log of the total number of alerts 
received by that physician during the same period, in a Poisson 
regression. Correlations across time were modeled as a 'spatial 
power' process (equivalent to an autoregressive 1 process). While 
CTAs could have triggered more than once for a given patient- 
physician pair if the initial CTA was ignored and if the patient 
returned during the study period and was still eligible for the 
alert, any such subsequent alerts were not included in our 
analyses. We used 2-week time periods for our analyses, for 
a total of 18 time periods over this 36-week study. Results are 
presented in numeric and graphical form along with the relevant 
p values to indicate the significance of the findings. 

RESULTS 

During the 36-week CTA exposure period, 915 total alerts were 
triggered for 178 physicians in the intervention arm of the 
associated randomized controlled study of the CTA. Eight alerts 
were discarded because they represented second alerts for 
patients who had earlier received an alert during the study 
period that was not responded to, leaving 907 eligible alerts for 
analysis. 

During the initial time period, the response rate to CTAs was 
about 50% among all users but dropped significantly over time 
by 2.7% for each advancing 2-week period, and that trend was 
significantly different from zero (flat) (p<0.0001; figure 2). 
Notably, there was still a 35% response rate at the 36th week of 
exposure. 

Subgroup analyses of response rate changes over time reveal 
that there are no significant differences between subspecialists 
(neurologists in this case) and generalists, with both groups 
trending downward to a similar extent, whether or not 



Figure 1 Screen shot of the clinical 
trial alert used in the randomized 
controlled trial that was the basis for 
the current study. 
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Figure 2 Physician response rates to 
clinical trial alerts (CTAs) are plotted at 
2-week intervals over the 36-week 
study. The solid line tracks response 
rates at each time point. The dashed 
line represents the linear regression line 
through each time point. Response 
rates declined at a rate of 2.7% per 
2-week time period (p<0.0001). 
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community generalists are included in the analyses. However, 
there is a greater response rate drop-off among all community- 
based providers compared with all university-based physicians 
(p=0.0489). Further, that drop-off is stronger when university- 
based subspecialists (neurologists) are removed and the analyses 
are restricted to generalists in both types of settings (p=0.0146). 

Referrals rates started at about 33% and, although they fluc- 
tuated, declined to about 9% by the end of the study period. Of 
note, generalists made fewer referrals than subspecialists 
(p<0.0001), and community-based physicians made fewer 
referrals than university-based physicians (p=0.006). The decline 
in referrals rates over time was more pronounced than the 
decline in response rates noted above. Specifically, there was 
a significant 4.9% decrease in referral rates per time period 
(p=0.0294) (figure 3). 

While absolute referral rates differed between groups, 
subgroup analyses of physician-generated CTA referral rates 
revealed no significant differences in rate declines between: (1) 
subspecialists versus generalists; (2) community-based versus 
university-based physicians; and (3) community-based versus 
university-based generalists only (ie, excluding neurologists). 

DISCUSSION 

The use of EHR-based CTAs has been demonstrated to increase 
participant recruitment rates to clinical trials, and is a promising 
approach for overcoming the major problem of inadequate and 
slow participant recruitment.^^' Because such an approach will 
necessarily be employed in the context of complex and varied 
clinical care environments, information on the performance 
characteristics and response patterns among different groups of 
potential end-users is needed to inform its application and use. 

These findings add to our understanding of how such alerts 
for clinical trials operate in real-world implementations by 



demonstrating empirically how and to what extent the rates of 
responses to such alerts decline across a variety of settings and 
end-users. Notably, they reveal, as hypothesized, that responses 
to point-of-care CTAs decline over time, although not as 
severely as anticipated, at least with regard to response rates. 

Indeed, overall response rates to this series of alerts was 
initially high at 50% and remained reasonably high at 35% even 
after 36 months of exposure, compared to CDS alerts which 
tend to have 4%— 51% response rates. ■'^ While the fall in 
response rate suggests alert fatigue over time, the fact that 
a substantial proportion of the alerts were still being responded 
to at 36 weeks suggests that such a duration of use may still 
provide benefit. However, the finding that referral rates declined 
more quickly and more precipitously over time than response 
rates suggests there might be a point after which use of a CTA 
might not be worth even the minimal disruption they cause. 
In addition, the differences seen among community-based versus 
university-based physicians suggest that future CTA deploy- 
ments should be tailored to a particular setting (ie, shorter in 
community-based settings and longer in university-based 
settings) in order to maximize benefit while avoiding excess 
fatigue. Additionally, as noted with some CDS alerts, tailoring of 
the alert's operating characteristics (eg, increasing specificity 
such that they trigger less often) might also affect response 
patterns and ultimately effectiveness, particularly in practice 
settings or specialties where response rates fall more rapidly. ™ 

While the design of this study does not allow for definitive 
determination of the reasons for the declines noted, the differ- 
ence between the response rate decline (2.7% per time period) 
and the referral rate decline (4.9% per time period) might reflect 
the fact that the act of CTA referral requires more effort than 
a simple response, and therefore causes more fatigability over 
time. However, this difference could also suggest the presence of 



Figure 3 Physician-generated referral 
rates using clinical trial alerts (CTAs) 
are plotted at 2-week intervals over the 
36-week study. The solid line tracks 
referrals rates at each time point. The 
dashed line represents the linear 
regression line through each time point. 
Referral rates declined at a rate of 4.9% 
per 2-week time period (p=0.0294). 
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other factors such as the possibiUty that declines in referrals 
reflect a drop in the available pool of eligible or interested 
candidates rather than alert fatigue. However, the population of 
potentially eligible participants (ie, patients with a recent 
stroke) remained relatively constant during the study, making 
this less likely. Nevertheless, it is probable that the reasons for 
the declines were multi-factorial, reflecting the combined influ- 
ence of alert fatigue and other factors. Additional studies, 
including qualitative studies to assess physician-user percep- 
tions, are ongoing and should help clarify other reasons for the 
declines noted. 

Comparison of physician response patterns over time and 
apparent alert fatigue with those when similar CDS approaches 
are employed for clinical use would be useful. Unfortunately, 
data on such changes over time, in CDS response rates appear to 
be lacking in the published literature. As noted above, plentiful 
circumstantial evidence of this aspect of alert fatigue in many 
studies reveals less than ideal average rates of response to CDS 
interventions,^^ with some studies commenting on the 
common behavior of overriding alerts,^" and still others 
addressing changes that can increase average response rates by 
improving the usability or appropriateness of alerts. However, 
although this form of alert fatigue over time undoubtedly exists, 
there has been surprisingly little empirical evidence of it, or data 
to characterize the nature of the phenomenon. Our study 
appears to be among the first to empirically demonstrate this 
aspect of alert fatigue by tracking changes in clinician response 
to alerts over time. Therefore, we believe it has implications 
beyond recruitment using CTAs, and that such an approach to 
measuring responses over time can help advance understanding 
of alert fatigue in general. We also believe that the methodology 
employed here could be used to evaluate and refine the design 
and application of decision support alerts in the future. 

Although the randomized study design and multi-user, multi- 
environment setting strengthen these findings and advance our 
understanding of CTA usage, this study has some limitations. 
These findings were derived from a single study of CTAs 
employed m a single trial of patients with recent stroke. 
Whether these findings would differ if the CTA were applied to 
another type of trial or in different settings remains to be 
determined. Also, while the CTA approach has been demon- 
strated to be effective using multiple EHR platforms,^^' this 
study employed a single EHR and these findings might differ 
with the use of another EHR. Furthermore, this alert was 
employed in a setting where other alerts were rarely triggered. 
Another factor possibly impacting response rates over time is 
the threshold setting (ie, sensitivity vs specificity) for a given 
alert. Whether the findings of this study would differ if there 
were multiple or more frequent alerts is not known but is 
possible given that multiple simultaneous alerts are a commonly 
cited factor leading to alert fatigue as noted above, and should be 
studied. 



CONCLUSION 

Physician response rates to CTAs started and remained relatively 
high even after a period of use, although they gradually but 
significantly declined over time. While overall response rates 
were lower among generalists than subspecialists, the rates of 
decline in CTA responses and referrals varied significantly only 
between university-based versus community-based physicians, 
and not between generalists versus subspecialists. These data 
also suggest that alert fatigue over time is likely a factor that 
must be taken into account when CTAs are employed. 



While it is currently unclear how the nature and degree of 
alert fatigue for CTAs compares to that of other types of CDS 
alerts, this study has implications for the implementation and 
management of such alerts. The methodology used here also 
appears to have implications for studies into the relative impact 
of alert fatigue across a range of decision support alert inter- 
ventions. Overall, these findings offer much-needed empirical 
data about the performance characteristics of CTAs, data that 
should help inform the tailoring and application of CTAs in real- 
world environments in order to overcome the major research 
challenge of improving and accelerating participant recruitment. 
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